The present invention relates to the fields of face recognition and gesture detection.
Head pose estimation is generally addressed as a three-dimensional problem, where the studied parameters concern the three ways of rotating a head. The main uses of estimating head poses concern the man-machine interface, face recognition, teleconferencing for low bit-rate transmission, virtual reality, and the construction of a three dimensional model of a face to simulate and display synthetic facial expressions. Conventional efforts have focused principally on developing hardware-based, three-dimensional (3D) image acquisition systems. Basically, such systems obtain 3D information using two different images of the same object. In the case of stereovision, two calibrated cameras are used. The society Eyetronics(copyright) has proposed ShapeSnatcher(copyright), a low-cost 3D-acquisition system that uses a single camera coupled with a slide projector. A grid is projected onto the object and distortion of the observed grid is composed with the original grid. In this manner, 3D information can be reconstructed.
A number of patents deal with various aspects of head pose estimation including face location, face tracking, facial feature location, detection of eyes, detection of lips, face recognition, age classification of human faces, facial animation, facial motion tracking, synthesis of facial expressions for toys or figurine, and the like. Except for face location or face tracking, where the face is considered as a region of the image verifying given properties, the other systems consider the face to be acquired in a frontal view, like a mug-shot photo.
An important challenge for face recognition systems is to recognise a face under varying poses. There are two streams of research in this field. The first one tries to transform (using image processing) off-frontal views of faces into frontal views and applies well-known frontal face recognition algorithms. The second stream of research tries to identify the pose of the face using a recognizer trained on a specific pose(s).
Hattori, K., Matsumori, S., and Sato, Y., xe2x80x9cInternational Conference on Pattern Recognitionxe2x80x9d, 1998, pp 1183-1187, Brisbane, describe an analog system. The hardware-based face measurement system uses a light source in a dark room, a color CCD camera, and two laser scanners on both sides of the camera. The system obtains a precise 3D model of the face, and head pose estimation can be computed.
A number of publications deal with head pose estimation algorithms, largely based on 3D image acquisition devices. One publication on face pose detection uses a single grey level image and neural networks to classify face images into a few categories: left, frontal, and right. One publication describes a system of pose discrimination based on the support vector machine using a single grey-level image. This approach uses a statistical method, based on a multi-dimensional linear classification, used on the whole face image. The output is a qualitative response among the three poses: xe2x80x9cleftxe2x80x9d, xe2x80x9cfrontalxe2x80x9d, xe2x80x9crightxe2x80x9d. However, this system does not provide a quantitative measure of face direction, does not provide vertical directions, and cannot be further extended for facial gesture recognition.
Systems for transforming facial images from off-frontal to frontal include (1) Feng, G. C Yuen, P. C. and Lai, J. H., xe2x80x9cA New Method of View Synthesis under Perspective Projectionxe2x80x9d, Proceedings of the 2nd International Conference on Multi-modal Interfaces, pp IV-96, IV-101, Hong-Kong 1999, and (2) Maruyama, M., Asano, S., and Nakano, Y., xe2x80x9cFace Recognition by Bi-directional View Synthesisxe2x80x9d, Proceedings of the 14th International Conference on Pattern Recognition, pp 157-159, Brisbane, Australia, 1998. Each of these documents describes methods for transforming off-frontal faces into frontal faces. Each of these techniques aims is to improve facial recognition performance alone. Disadvantageously, each of the documents teaches transforming an image before facial recognition is carried out on the image. Such transformations create a noisy image, as may be observed from the results listed obtained in each document, leading to errors in facial recognition.
Systems for 3-D pose estimation using a single camera are described by each of (1) Park, K. R., Nam, S. W., Han, S. C. and Kim, J., xe2x80x9cA Vision-Based Gaze Detection Algorithmxe2x80x9d, Proceedings of the 2nd International Conference on Multi-modal Interfaces, pp IV-47, IV-50, Hong-Kong 1999, (2) Chen, Q., Wu, H., Shioyama, T., Shimada, T. and Chihara, K., xe2x80x9cA Robust Algorithm for 3D Head Pose Estimationxe2x80x9d, Proceedings of the 14th International Conference on Pattern Recognition, pp 1356-1359, Brisbane, Australia, 1998, and (3) Wu, H., Shioyama, T. and Kobayashi, H., xe2x80x9cSpotting Recognition of Head Gestures from Color Images Seriesxe2x80x9d, Proceedings of the 14th International Conference on Pattern Recognition, pp 83-85, Brisbane, Australia, 1998. Each of these documents describes 3D head pose estimation using a single camera. In each document, skin color and hair color regions are generally used to establish the 3D-parameters. Information about the skin and the hair color region is used to estimate the pose of the head, because the two types of information can be robustly extracted from color images and both are not sensitive to the changes of facial expression, wearing glasses, and other local changes of facial features. A skin region is represented with the Skin Color Distribution Model, and a hair region is represented with the Hair Color Distribution Model.
Secondly, the foregoing systems each approximate the head as a 3D-elliptical sphere and compute primary and secondary moments of these two regions to derive the 3D-head pose. The tilt is just the principal axis of the face region, and the two other rotations (xe2x80x9cup-downxe2x80x9d, xe2x80x9cleft-rightxe2x80x9d) are expressed using third order polynomials depending on the horizontal and vertical distance of the centroid of the face region with the centroid of the skin region. These two polynomials and their specific coefficients have been learnt statistically and are fixed. Disadvantageously, these methods require a crucial segmentation step. Segmentation can fail easily in certain circumstances such as complex background or for a person without hair. Accordingly, each of these techniques has significant limitations and is not robust. Kuno, Y., Adachi, Y., Murashima, T., and Shirai, Y., xe2x80x9cIntelligent Wheelchair Looking at Its Userxe2x80x9d, Proceedings of the 2nd International Conference on Audio-and Video-based Biometric Person Authentication, pp-84-89, Washington, 1999, describe a system for determining face direction using a tracking algorithm. This system involves the detection of the face direction to guide an intelligent wheelchair, which is proposed as an alternative to the computation of the face direction. In this system, the face is tracked. If the face turns, the wheelchair turns. Precise computation of the degree of face turning is not required and they do not consider xe2x80x9cup-downxe2x80x9d movement, because the camera is always below the face of the user. The system detects roughly the eyes, the eyebrows, the nostrils, and the two mouths corners. These points are materialized by square regions and, to compute the face direction, the relative position of these squares is compared with squares of the same frontal face. Moreover, these squares are tracked throughout sequence of images, stabilizing their extraction. The system uses several frames to decide the face direction and does not give a quantitative result.
Huang, J., Shao, X., and Wechsler, H., xe2x80x9cFace Pose Discrimination Using Support Vector Machine (SVM)xe2x80x9d Proceedings of the 14th International Conference on Pattern Recognition, pp 155-156, Brisbane, Australia, 1998, describes a system of pose discrimination based on the support vector machine using a single gray level image. This approach uses a statistical method based on a multi-dimensional linear classification, used on the whole face image. The output is a qualitative response among the three poses: xe2x80x9cleftxe2x80x9d, xe2x80x9cfrontalxe2x80x9d, xe2x80x9crightxe2x80x9d. However, this system does not provide a quantitative measure of face direction, does not provide vertical directions, and cannot be further extended for facial gesture recognition.
In the field of the video conference systems, several patents including U.S. Pat. Nos. 5,500,671, 5,359,362, and 5,675,376, and publications deal with the gaze detection and the line of sight estimation of the conferees. The purpose is to send to the conferee, the face image of an interlocutor, permitting communications with eye-to-eye contact. In the videoconference systems, conventional methods consider that the face is frontal and make intensive use of the gaze detection to provide eye-to-eye communications between the conferees. Generally speaking, the systems seek to ensure that the faces transmitted through the communication networks looks satisfying. Using the face direction estimation, more than the gaze detection, it is possible to enhance the quality of the transmitted images by generating a frontal view when the face is off-frontal, or by sending a pre-registered frontal face, or by unchanging the current satisfying frontal face which is displayed.
Thus, a need clearly exist for an improved system of recognizing faces from single images and for determining facial gestures.
The aspects of the invention enable, given a single digital image of a person""s face, the determination of a face direction by computing a nose axis. The nose axis maximizes a correlation measure between left and right sides of the face. The correlation measure is computed by comparing one of the two sides with another synthetic side derived from the other side using symmetry and perspective transforms on the second side. Optionally, this process can be accelerated using a contrast enhancement algorithm, which takes advantage of the circumstances that the nose is the part of the face between the eyes which reflects the most light and that this reflected light is represented as a line-like region close to the real nose axis. Preferably, the result of this computation is a word describing the spatial position of the face and combining height (xe2x80x9cupxe2x80x9d, xe2x80x9cnormalxe2x80x9d, xe2x80x9cdownxe2x80x9d) and neck-rotation (xe2x80x9cleftxe2x80x9d, xe2x80x9cfrontalxe2x80x9d, xe2x80x9crightxe2x80x9d).
Further, for a sequence of images, direction changes and facial gestures are detected by tracing the face direction. Still further, face recognition accuracy can be improved using face direction detection.
In another aspect of the invention, a visual mouse is provided using the face direction detection method as an interface for applications, such as interactive games, training, and the like.
In yet another aspect of the invention, there is provided a video conferencing method providing eye-to-eye contact communications using the face direction to enhance the quality of facial images.